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MANAGEMENT) 

DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 

SUBJECT :  Report  on  the  Hotline  Complaint  on  Management  of  the  Cooperative 
Engagement  Capability  Program  (Report  No.  95-143) 


We  are  providing  this  report  for  your  review  and  comments.  This  audit  resulted 
from  a  complaint  to  the  DoD  Hotline  concerning  the  Navy's  management  of  the 
Cooperative  Engagement  Capability  Program.  We  received  comments  on  a  draft  of 
this  report  from  only  the  Director,  Operational  Test  and  Evaluation.  We  considered 
those  comments  in  preparing  the  final  report.  Based  on  those  comments,  we  revised 
Recommendation  B.2.  and  redirected  the  recommendation  to  the  Assistant  Secretary  of 
the  Navy  (Research,  Development  and  Acquisition). 

DoD  Directive  7650.3  requires  that  all  audit  recommendations  be  resolved 
promptly.  Therefore,  we  request  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition);  the  Director,  Test,  Systems  Engineering  and 
Evaluation;  the  Commander,  Naval  Sea  Systems  Command;  and  the  Program  Manager, 
Cooperative  Engagement  Capability  Program,  provide  final  comments  on  the 
recommendations  by  May  9,  1995.  The  recommendations  are  subject  to  resolution  in 
accordance  with  DoD  Directive  7650.3  in  the  event  of  nonconcurrence  or  failure  to 
comment.  We  also  ask  that  your  comments  indicate  concurrence  or  nonconcurrence 
with  the  material  internal  control  weaknesses  highlighted  in  Part  1. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  If  you  have  any 
questions  on  this  report,  please  contact  Mr.  John  E.  Meling,  Program  Director,  at 
(703)  604-9091  (DSN  664-9091)  or  Mr.  Jack  D.  Snider,  Project  Manager,  at 
(703)  604-9087  (DSN  664-9087).  Appendix  G  lists  the  distribution  of  this  report.  The 
inside  back  cover  lists  the  audit  team  members. 


Assistant  Inspector  General 
for  Auditing 
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(Project  No,  4AE-8005) 

HOTLINE  COMPLAINT  ON  MANAGEMENT  OF  THE 
COOPERATIVE  ENGAGEMENT  CAPABILITY  PROGRAM 

EXECUTIVE  SUMMARY 

Introduction.  This  audit  resulted  from  a  complaint  to  the  DoD  Hotline  that  the  Navy 
was  not  effectively  managing  the  Cooperative  Engagement  Capability  (CEC)  Program. 
The  CEC  Program,  an  acquisition  category  IC  effort,  is  intended  to  improve  the 
Navy's  anti-air  warfare  capability  by  coordinating  all  anti-air  warfare  sensors  into  a 
single  real-time,  fire  control,  radar  picture.  The  CEC  Program  began  in  FY  1985  and 
was  a  classified  program  until  December  1993.  While  technically  not  the  prime 
contractor  for  the  CEC  Program,  the  Johns  Hopkins  University/Applied  Physics 
Laboratory  (the  Laboratory)  oversees  engineering  and  development  for  the  Program. 
The  Navy  estimated  the  CEC  Program  costs  to  be  $2.5  billion  for  research, 
development  and  procurement  in  FY  1993  dollars.  In  March  1995,  the  Navy  plans  to 
hold  a  Milestone  II,  Development  Approval,  decision  for  the  CEC  Program.  The  Navy 
plans  to  declare  attainment  of  initial  operational  capability  (IOC)  for  the  CEC  Program 
in  late  FY  1996  before  operational  testing  and  a  Milestone  III,  Production  Approval, 
decision. 

Objectives.  The  audit  objective  was  to  evaluate  the  DoD  Hotline  allegations 
concerning  the  CEC  Program  and  the  effectiveness  of  the  milestone  review  process. 
We  assessed  the  adequacy  of  the  acquisition  strategy  and  program  documentation, 
including  information  in  support  of  major  milestone  and  program  reviews.  The  audit 
also  evaluated  the  adequacy  of  internal  controls  related  to  the  objective. 

Audit  Results.  The  results  of  our  review  substantiated  11  of  the  13  Hotline  assertions 
and  allegations.  Specifically,  the  CEC  Program  Office: 

o  had  not  established  adequate  controls  to  manage  and  develop  the  CEC 
Program  effectively.  As  a  result,  the  CEC  Program  Office  could  not  ensure  that  the 
Laboratory  provided  fair  and  reasonable  contract  prices  and  could  not  oversee  and 
measure  the  Laboratory's  performance  against  contract  cost,  schedule,  and  performance 
requirements  (Finding  A). 

0  was  not  planning  to  operationally  test  production  CEC  equipment  before 
fielding  it.  As  a  result,  the  Navy  may  declare  the  attainment  of  IOC  for  the  CEC 
Program  based  on  an  operational  assessment  of  a  prototype  before  decisionmakers  have 
evidence  that  the  system  will  meet  user  requirements  and  is  operationally  effective  and 
suitable  (Finding  B). 

Internal  Controls.  The  audit  identified  material  internal  control  weaknesses.  The 
Navy  did  not  implement  necessary  internal  controls  to: 

o  manage  the  Laboratory's  cost,  schedule,  and  performance  in  the  CEC 
contract  and 

o  verify  the  CEC  Program's  operational  effectiveness  and  suitability  through 
operational  testing  before  the  planned  IOC  date. 


We  identified  those  weaknesses  even  though  the  CEC  Program  Office  identified  DoD 
Internal  Management  Control  Program  assessable  units  and  conducted  vulnerability 
assessments  of  the  CEC  Program  in  July  1992  and  August  1994.  The  Program  Office 
assigned  a  low  overall  vulnerability  rating  in  those  assessments.  Internal  controls 
assessed  are  summarized  in  Part  I  of  this  report. 

Potential  Beneflts  of  Audit.  We  could  not  quantify  the  monetary  benefits  to  be 
realized  by  implementing  the  recommendations  but  system  effectiveness  and  readiness 
for  production  would  be  improved.  Implementation  of  the  recommendations  will 
ensure  that  the  CEC  Program  Office: 

o  obtains  fair  and  reasonable  prices  on  the  CEC  contract  from  the  Laboratory; 

o  establishes  CEC  contract  cost,  schedule,  and  performance  controls;  and 

o  declares  the  CEC  Program's  IOC  date  if  operational  test  results  show  that  the 
CEC  Program  is  operationally  effective  and  suitable  (Appendix  E). 

Summary  of  Recommendations.  We  recommend  that  the  Commander,  Naval  Sea 
Systems  Command  award  a  separate  CEC  contract  for  the  engineering  and 
manufacturing  development  phase  of  the  acquisition  process  and  incorporate  cost  and 
pricing  data  requirements  into  the  contract.  We  also  recommended  that  the  CEC 
Program  Manager  submit  the  Test  and  Evaluation  Master  Plan  to  the  Director,  Test, 
Systems  Engineering  and  Evaluation,  for  review  and  approval;  that  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  and  Acquisition)  request  Congress  to 
extend  the  IOC  date  so  that  the  attainment  of  IOC  can  be  based  on  an  operational  test 
of  production  equipment;  and  that  the  Director,  Test,  Systems  Engineering  and 
Evaluation,  evaluate  the  system's  developmental  testing  results  before  the  system  is 
allowed  to  proceed  with  the  dedicated  phase  of  operational  testing  to  support  the 
plaimed  production  decision. 

Management  Comments.  We  received  comments  to  a  draft  of  this  report  from  the 
Director,  Operational  Test  and  Evaluation.  However,  we  did  not  receive  comments 
from  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology  and 
the  Navy.  The  Director,  Operational  Test  and  Evaluation,  stated  that,  procedurally,  his 
approval  of  the  Test  and  Evaluation  Master  Plan  is  based  on  provisions  in  the  Plan  for 
a  dedicated  operational  test  and  evaluation  to  support  a  beyond  low-rate  initial 
production  and  not  a  declaration  of  the  attainment  of  IOC.  Therefore,  his  office  did 
not  have  the  authority  to  withhold  approval  of  the  Test  and  Evaluation  Master  Plan 
unless  it  includes  the  performance  of  a  dedicated  operational  test  and  evaluation  of  the 
CEC  Program  before  declaring  the  attainment  of  IOC.  Part  II  contains  a  discussion  of 
the  Director's  comments  and  Part  IV  contains  the  complete  text. 

Audit  Response.  Based  on  the  Directors'  comments,  we  revised  and  redirected  the 
recommendation  to  the  Navy.  Since  the  Navy  is  basing  the  attainment  of  IOC  on  an 
operational  assessment  of  prototype  CEC  equipment,  rather  than  on  an  operational  test 
of  production  CEC  equipment,  to  meet  a  congressionally  mandated  IOC  in  FY  1996, 
the  Navy  should  request  Congress  to  extend  the  IOC  date  so  that  the  IOC  can  be  based 
on  an  operational  test  of  production  CEC  equipment.  We  request  that  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  and  Acquisition);  the  Director,  Test, 
Systems  Engineering  and  Evaluation;  the  Commander,  Naval  Sea  Systems  Command; 
and  the  Program  Manager,  Cooperative  Engagement  Capability  Program,  provide 
comments  on  the  recommendations  in  the  final  report  by  May  9,  1995. 
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Part  I  -  Introduction 


Introduction 


Background 


Hotline  Allegations.  The  DoD  Hotline  received  a  complaint  that  the  Navy  had 
not  established  adequate  controls  to  manage  the  Cooperative  Engagement 
Capability  (CEC)  Program  effectively.  Appendix  A  of  this  report  addresses 
each  of  the  13  Hotline  assertions  and  allegations,  and  the  results  of  our  review. 

Cooperative  Engagement  Capability  Program.  The  CEC  Program  is  an 
acquisition  category  (ACAT)  IC  program  that  the  Navy  managed  as  a  classified 
program  until  December  1993.  (See  Appendix  B  for  definitions  of  terms  in  this 
report.)  The  CEC  Program  is  intended  to  improve  battle  force  anti-air  warfare 
capability  by  coordinating  all  Naval  force  radars  into  a  single  real-time,  fire 
control,  radar  picture.  The  CEC  Program  requires  an  E-2C  "Hawkeye"  aircraft 
to  maximize  the  over-the-horizon  distance  between  ships  and  to  compensate  for 
the  curvature  of  the  earth  and  the  effects  of  weather  conditions.  Additionally, 
Congress  wants  the  Army  and  the  Air  Force  to  incorporate  the  CEC  Program 
into  specific  weapon  systems,  such  as  the  Army’s  Patriot  Air  Defense  Guided 
Missile  System  and  the  Air  Force's  E-3  Airborne  Warning  and  Control  System 
aircraft. 

Concept.  In  the  mid-1980s,  the  Johns  Hopkins  University /Applied 
Physics  Laboratory  (the  Laboratory)  developed  the  CEC  concept.  In  September 
1985,  the  Office  of  the  Chief  of  Naval  Operations  approved  the  concept 
exploration  phase  and,  in  FY  1989,  the  demonstration  and  validation  phase  for 
the  CEC  Program.  A  milestone  program  review  and  decision  did  not  precede 
either  of  those  phases;  however,  die  Navy  has  scheduled  a  Milestone  II, 
Development  Approval,  decision  for  March  1995.  The  Navy  plans  to  declare 
attainment  of  initial  operational  capability  (IOC)  in  late  FY  1996  but  will  not 
conduct  operational  testing  until  FY  1999. 

Development  Effort.  The  CEC  Program  is  comprised  of  four  primary 
development  efforts:  the  Cooperative  Engagement  Processor  (the  CEC 
processor),  the  Data  Distribution  System  (DDS),  the  Common  Equipment  Set, 
and  systems  integration  with  existing  Navy  weapon  systems.  The  Laboratory  is 
developing  the  CEC  processor  software  and  managing  systems  integration. 
E-Systems,  ECI  Division,  is  developing  the  CEC  processor  hardware,  the  DDS, 
and  the  Common  Equipment  Set  and  performing  systems  integration.  The  Navy 
estimated  the  Program  will  cost  $2.5  billion  for  research,  development,  and 
procurement  in  FY  1993  dollars.  The  Navy  plans  to  procure  178  Common 
Equipment  Sets  comprised  of  121  shipboard  units,  47  airborne  units,  and 
10  land-based  test  sites. 

Johns  Hopkins  University/Applied  Physics  Laboratory.  The 
Laboratory,  an  educational  research  and  development  facility,  is  the  CEC 
Program's  technical  direction  agent  and  design  agent.  The  Laboratory,  as  the 
technical  direction  agent,  provides  Program  technical  direction  beyond  the 
expertise  of  the  Naval  Sea  Systems  Command  CEC  Program  Office.  In  this 
role,  the  Laboratory  provides  Program  direction  to  all  CEC  contractors  and 
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subcontractors.  The  Laboratory  is  the  technical  direction  agent  for  the  CEC 
Program  as  well  as  the  design  agent  for  the  CEC  processor  and  is  developing 
the  CEC  processor  software. 

Omnibus  Contracts.  The  Space  and  Naval  Warfare  Systems  Command 
(SPAWAR)  has  an  omnibus  cost-plus-fixed-fee  research  and  development 
contract,  N00039-91-C-0001,  with  the  Laboratory.  Under  this  omnibus 
contract,  the  Laboratory  performs  the  CEC  development  effort.  From 
December  1990  through  September  1994,  the  Navy  provided  the  Laboratory 
with  about  $144.8  million  toward  the  development  of  the  CEC  Program.  The 
Navy  awarded  this  contract  on  a  sole-source  non-competitive  basis.  The 
contract  expired  September  1994.  In  November  1994,  SPAWAR  awarded  a 
new  omnibus  contract  for  about  $425  million  to  the  Laboratory.  The  contract 
will  include  two  option  years  for  about  $400  million  each  that  will  result  in  a 
cumulative  contract  total  of  about  $1.2  billion.  Of  this  $1.2  billion,  SPAWAR 
estimated  that  about  $72  million  will  be  for  engineering  and  manufacturing 
development  of  the  processor  software  for  the  CEC  Program.  The  CEC 
Program  Office  estimated  that  it  will  require  an  additional  $96  million  to 
complete  engineering  and  manufacturing  development  of  the  CEC  processor 
software.  For  the  CEC  effort,  SPAWAR  personnel  perform  the  contract 
functions  of  the  procuring  contracting  officer  (PCO),  the  contracting  officer's 
representative  (COR),  and  the  administrative  contracting  officer. 

E-Systems  Contracts.  The  CEC  Program  Office  awarded  E-Systems  two  sole- 
source  non-competitive  cost-plus-award-fee  contracts  for  demonstration  and 
validation  of  the  CEC  processor  hardware  and  the  DDS.  On  September  30, 
1988,  the  Program  Office  awarded  contract  N00024-88-C-5256,  totaling 
$72.0  million,  that  was  effective  through  February  20,  1994.  On  June  1,  1992, 
the  Program  Office  awarded  contract  N00024-92-C-5230,  totaling 
$138.4  million,  that  is  effective  through  January  30,  1996;  however,  as  of 
September  29,  1994,  only  $11  million  remained  available  for  obligation.  After 
the  Milestone  II  decision  in  March  1995,  the  CEC  Program  Office  plans  to 
award  E-Systems  a  sole-source,  cost-plus-award-fee  contract,  totaling  about 
$292  million,  for  engineering  and  rnanufacmring  development  of  the  CEC 
processor  hardware,  the  DDS,  and  the  Common  Equipment  Set.  The  Naval  Sea 
Systems  Command  administers  the  E-Systems  contracts. 


Objectives 


The  audit  objective  was  to  evaluate  the  DoD  Hotline  allegations  concerning  the 
CEC  Program  and  the  effectiveness  of  the  milestone  review  process.  We 
psessed  the  adequacy  of  the  acquisition  strategy  and  program  documentation, 
including  information  provided  in  support  of  major  milestone  and  program 
reviews.  The  audit  also  evaluated  the  adequacy  of  internal  controls  related  to 
the  objective. 
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Scope  and  Methodology 


We  conducted  the  program  audit  from  February  1994  through  November  1994 
and  reviewed  data  dated  from  June  1989  toough  November  1994.  To 
accomplish  the  objective,  we: 

o  reviewed  the  13  DoD  Hotline  assertions  and  allegations  to  determine 
their  validity, 

o  examined  the  CEC  portions  of  contract  N00039-91-C-0001  with  the 
Laboratory, 

o  examined  contracts  N00024-88-C-5256  and  N00024-92-C-5230  with 
E-Systems, 

o  evaluated  the  effectiveness  of  the  milestone  review  and  program 
review  process  for  the  CEC  program,  and 

o  discussed  issues  relating  to  the  effectiveness  of  the  CEC  milestone 
review  process  with  personnel  from  the  Office  of  the  Secretary  of  Defense 
(OSD). 

We  conducted  the  audit  in  accordance  with  auditing  standards  issued  by  the 
Comptroller  General  of  the  United  States,  as  implemented  by  the  Inspector 
General,  DoD,  and  accordingly  included  such  tests  of  internal  controls  as  we 
deemed  necessary.  We  did  not  rely  on  computer-processed  data  to  support  our 
findings  and  recommendations  because  the  areas  reviewed  did  not  contain 
computer-processed  data.  Appendix  F  lists  the  organizations  visited  or 
contacted. 


Internal  Controls 


Internal  Controls  Evaluated.  We  evaluated  internal  controls  related  to  the 
effectiveness  of  the  milestone  review  process  and  the  adequacy  of  the 
information  provided  to  the  milestone  decision  authorities  in  support  of  major 
milestone  and  program  reviews  for  the  CEC  Program.  The  DoD  Directive 
5000.1,  "Defeiise  Acquisition,"  February  23,  1991;  DoD  Instruction  5000.2, 
"Defense  Acquisition  Policies  and  Procedures,"  February  23,  1991;  and  DoD 
Manual  5000. 2-M,  "Defense  Acquisition  Management  Documentation  and 
Reports,"  February  23,  1991,  specify  those  controls  and  procedures.  We  also 
assessed  implementation  of  the  requirements  of  DoD  Directive  5010.38, 
"Internal  Management  Control  Program,"  April  14,  1987,  including 

performance  of  vulnerability  assessments  and  management  control  reviews. 

Internal  Control  Weaknesses  Identifred.  We  identified  material  internal 
control  weaknesses  as  defined  by  DoD  Directive  5010.38.  Specifically,  the 
Navy  did  not  implement  internal  controls  necessary  to: 
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o  manage  the  Laboratory's  cost,  schedule,  and  performance  in  the  CEC 
contract  and 

o  verify  the  CEC  Program's  operational  effectiveness  and  suitability 
through  operational  testing  before  the  planned  initial  operational  capability  date. 

We  identified  those  weaknesses  even  though  the  CEC  Program  Office  identified 
DoD  Internal  Management  Control  Program  assessable  units  and  conducted 
vulnerability  assessments  of  the  CEC  Program  in  July  1992  and  August  1994. 
The  Program  Office  assigned  a  low  overall  vulnerability  rating  to  those 
assessments. 

Internal  Control  Weaknesses  Correction.  The  recommendations  in  this 
report,  if  implemented,  will  correct  the  internal  control  weaknesses.  Monetary 
benefits  associated  with  the  implementation  of  our  recommendations  are  not 
quantifiable  because  benefits  will  depend  on  future  actions  by  OSD  and  the 
Navy.  We  will  provide  a  copy  of  the  final  report  to  the  senior  official 
responsible  for  internal  controls  in  the  Navy. 


Prior  Audits  and  Other  Reviews 


Since  1989,  neither  the  General  Accounting  Office  nor  the  Inspector  General, 
DoD,  have  issued  reports  directly  related  to  the  audit  objective.  However,  the 
Inspector  General,  DoD,  issued  a  report  concerning  the  noncompetitive  award 
of  a  contract  to  the  Laboratory  without  adequate  justification.  We  synopsized 
the  report  in  Appendix  C. 
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Finding  A.  Program  Management  and 
Development 

The  CEC  Program  Office  had  not  established  adequate  controls  to 
manage  and  develop  the  CEC  Program  effectively.  This  condition 
occurred  because: 

o  the  Space  and  Naval  Warfare  Systems  Command  (SPAWAR) 
contract  used  for  the  development  of  the  CEC  Program  was  not 
strucmred  for  the  development  and  management  of  an  acquisition 
category  IC  program,  and 

o  the  Laboratory  was  not  required  to  provide  cost  and  pricing 
data  for  each  task  or  significant  subcontract. 

As  a  result,  the  CEC  Program  Office  could  not  ensure  that  the 
Laboratory  provided  fair  and  reasonable  contract  prices  and  could  not 
oversee  and  measure  the  Laboratory's  performance  against  contract  cost, 
schedule,  and  performance  requirements. 


Background 


The  FAR,  subparts  1.6  and  15.8,  "Contracting  Authority  and  Responsibilities" 
and  "Price  Negotiation,"  respectively,  provide  guidance  concerning  contract 
oversight  and  cost  and  pricing  data.  The  DoD  Instruction  5000.2  provides 
guidance  concerning  systems  engineering  management,  quality  control,  software 
development,  and  cost/schedule  control  systems  for  AC  AT  IC  programs.  The 
Defense  Systems  Management  College  Mission  Critical  Computer  Resources 
Management  Guide  provides  guidance  concerning  the  use  of  software  metrics  in 
systems  development. 


Management  and  Development  Controls 


Our  review  of  the  Hotline  allegations  disclosed  that  the  CEC  Program  Office 
did  not  establish  controls  to  manage  and  develop  the  CEC  Program  effectively. 
(Appendix  A  discusses  the  allegations  and  our  audit  results  concerning  the 
allegations.)  The  Program  Office  did  not  obtain  cost  and  pricing  data 
supporting  CEC  omnibus  contract  taskings,  require  a  work  breakdown  structure 
and  a  detailed  statement  of  work,  acquire  systems  engineering  management  data 
on  the  CEC  omnibus  contract,  adequately  monitor  contractor  software 
development,  and  require  the  Laboratory  to  establish  a  cost/schedule  control 
system. 
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Cost  and  Pricing  Data.  The  PCO  was  unable  to  perfonn  cost  or  price  analyses 
on  Laboratory  cost  estimates  because  the  Program  Office  did  not  require  the 
Laboratory  to  provide  cost  and  pricing  data  for  each  task  or  significant 
subcontract.  The  FAR,  subpart  15.8,  requires  contractors  to  submit  cost  and 
pricing  data  for  the  award  of  any  negotiated  contract  (except  for  undefinitized 
actions  such  as  letter  contracts)  expected  to  exceed  $500,000.  Additionally,  any 
modification  of  a  negotiated  contract,  when  the  modification  involves  a  price 
adjustment  expected  to  exceed  $500,000,  requires  cost  or  pricing  data  regardless 
of  whether  the  original  contract  required  cost  and  pricing  data.  The  cost  or 
pricing  data  enables  the  Government  to  perform  cost  or  price  analysis  and 
ultimately  allow  the  Government  and  the  contractor  to  negotiate  fair  and 
reasonable  contract  prices.  From  January  1,  1991,  through  September  30, 
1994,  SPAWAR  awarded  the  Laboratory  seven  modifications  on  die  omnibus 
contract  in  the  form  of  administrative  letters  that  each  totaled  more  than 
$500,000  as  shown  in  the  following  table: 

Omnibus  Contract  Modifications  Exceedine  $500.000 


Administrative 

Dollar 

Letter 

Performance 

Value 

Number 

Period 

fin  thousands) 

13200-20 

01/01/91  through  12/31/91 

$  8,725 

13198 

02/01/91  through  12/31/91 

20,100 

13350-16 

01/01/92  through  12/31/92 

22,548 

13408 

02/01/92  through  12/31/92 

6,900 

13448 

06/01/92  through  12/31/92 

10,600 

13500-20 

01/01/93  through  09/30/93 

27,000 

13700-24 

10/01/93  through  09/30/94 

48.900 

$144,773 

Work  Breakdown  Structure.  The  work  breakdown  structure  is  the  foundation 
for  all  reporting  of  contractor  performance.  However,  the  CEC  Program  Office 
did  not  require  the  Laborato^  to  provide  a  work  breakdown  strucmre.  The 
DoD  Instruction  5000.2  requires  a  work  breakdown  structure  to  establish  the 
essential  framework  for  program  and  technical  planning,  cost  estimating, 
resource  allocations,  performance  measurement,  and  stams  reporting.  The 
Program  Office  asserted  that  the  Laboratory  followed  an  informal  work 
breakdown  structure;  however,  the  Laboratory  stated  that  it  did  not  prepare  or 
follow  a  work  breakdown  structure  for  the  CEC  Program. 

Detailed  Statement  of  Work.  A  statement  of  work  is  that  portion  of  a  contract 
that  establishes  and  defines  all  performance  requirements  for  contractors'  efforts 
either  directly  or  with  specific,  cited  documents.  Due  to  the  lack  of  a  clearly 
defined  statement  of  work,  CEC  contracting  personnel  could  not  determine 
whether  the  Laborato^  met  the  performance  requirements  necessary  for  the 
CEC  Program  to  achieve  its  objectives.  The  SPAWAR  used  administrative 
letters  to  serve  as  the  Laboratory's  statement  of  work.  However,  the  PCO  and 
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Defense  Contract  Audit  Agency  personnel  stated  that  the  information  in  the 
letters  did  not  clearly  define  the  Laboratory's  performance  requirements.  We 
agree. 

Systems  Engineering  Management.  The  Laboratory  and  E-Systems,  as  design 
agents,  are  responsible  for  overall  CEC  system  integration.  As  of  the  June  1994 
operational  assessment,  the  CEC  Program  Office  had  not  required  the 
Laboratory  to  prepare  and  follow  a  systems  engineering  management  plan  based 
on  a  completed  CEC  Program  operational  requirements  document.  The  DoD 
Instruction  5000.2  requires  program  offices  to  apply  systems  engineering 
throughout  the  program's  life-cycle  as  a  comprehensive  management  process  to 
translate  an  operational  need  into  a  configured  system,  integrate  the  technical 
inputs  of  the  entire  development  community,  and  ensure  the  compatibility  of  all 
functional  and  physical  interfaces.  Further,  the  systems  engineering 

management  plan  will  document: 

o  the  systems  engineering  processes, 

o  the  integration  of  required  technical  specialties, 

o  the  performance  reporting,  and 

o  the  key  engineering  milestones  and  schedules. 

Personnel  in  the  Office  of  the  Chief  of  Naval  Operations,  the  user's 
representative,  indicated  that  they  would  work  with  the  Laboratory  to  prepare 
the  operational  requirements  document  before  the  Milestone  II  decision. 
Previously,  however,  the  CEC  Program  Office  made  program  decisions  without 
a  systems  engineering  process  and  an  approved  systems  engineering 
management  plan. 

After  we  completed  our  audit  fieldwork  and  staffed  our  working  draft  report, 
the  CEC  Program  Office  provided  us  a  draft  systems  engineering  management 
plan,  dated  November  1994,  that  the  Program  Executive  Office,  Theater  Air 
Defense,  prepared. 

Quality  Control.  The  CEC  Program  Office  had  not  reviewed  the  Laboratory's 
quality  control  procedures  to  determine  whether  the  procedures  were  adequate. 
The  DoD  Instmction  5000.2  states  that  the  program  office  will  emphasize  and 
integrate  quality  control  throughout  the  program,  including  the  systems 
engineering  effort.  Quality  control  consists  of  the  quality  of  system  design, 
quality  of  conformance,  and  fitness  for  use. 

Software  Development.  The  CEC  Program  Office  did  not  require  the 
Laboratory  to  prepare  a  software  development  plan.  However,  subsequent  to 
the  completion  of  our  fieldwork  and  staffing  of  a  working  draft  report,  the  CEC 
Program  Office  provided  a  software  development  plan  for  the  CEC  processor, 
dated  October  1994.  The  DoD  Instruction  5000.2  requires  the  program  office 
to  manage  computer  resource  development  as  an  integral  part  of  the  overall 
program  development.  In  this  regard,  the  Instraction  requires  that  a  software 
development  plan  be  prepared  that  defines  the  software  processes,  including 
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corporate  policies,  practices,  and  standards  that  the  contractor  must  follow 
throughout  the  software  development  process.  Also,  the  program  office  is  to 
verify  that  the  developer  understands  the  scope  of  the  software  development 
effort  and  is  capable  of  meeting  user's  needs. 

Software  Metrics.  Software  metrics  are  documentation  showing  software  cost, 
schedule,  and  performance  objectives.  Additionally,  software  metrics  verify  the 
delivery  of  a  product  that  will  satisfy  the  user's  needs.  CEC  Program  Office 
persoimel  stated  that  the  CEC  processor  software  was  the  primary  component  of 
the  CEC  Program  to  which  all  other  components  of  the  CEC  Program  must 
conform.  However,  the  Program  Office  did  not  require  the  Laboratory  to 
provide  software  metrics  in  accordance  with  DoD  Directive  5000.2. 

After  we  completed  our  audit  fieldwork  and  staffed  our  working  draft  report, 
the  CEC  Program  Office  identified  several  documents  as  being  the  CEC 
software  metrics.  However,  those  documents  were  not  CEC  software  metrics 
but  summaries  of  trouble  status  reports  prepared  by  the  Naval  Surface  Warfare, 
Dahlgren  Division,  Dahlgren,  Virginia. 

Metrics  Requirement.  The  Defense  Systems  Management  College 
Mission  Critical  Computer  Resources  Management  Guide  states  that  program 
offices  should  use  software  metrics  to  balance  cost,  schedule,  and  performance 
objectives.  Software  metrics  are  divided  into: 

0  management  metrics,  which  indicate  progress  against  the  plan; 

0  quality  metrics,  which  effect  performance,  user  satisfaction, 
supportability,  and  ease  of  change;  and 

o  process  metrics,  which  provide  feedback  for  process 

improvement. 

Metric  Data.  The  CEC  Program  Office  stated  that  the  Laboratory  did 
not  have  the  level  of  sophistication  to  manage  multiple  software  baselines.  The 
lack  of  metrics  data  indicated  a  weakness  in  the  contractor's  ability  to  manage 
software  development  properly  and  to  provide  progress  visibility  to  the  program 
office.  The  CEC  Program  Office  plans  to  obtain  an  IOC  software  baseline  from 
the  Laboratory  without  the  benefit  of  metrics  data. 

Software  Capability  Evaluation.  The  Laboratory  evaluated  its  own 
software  capability.^  The  evaluation  disclosed  that  the  Laboratory  had  a 
weakness  in  requirements  generation  and  system  integration.  Regardless,  the 
CEC  Program  Office  did  not  require  the  Laboratory  to  provide  software  metrics 
so  the  Program  Oflfice  could  monitor  the  contractor's  requirements  generation 
and  system  integration.  CEC  Program  Office  personnel  stated  that  the 


^In  1984,  the  Air  Force  placed  the  Software  Engineering  Institute  under  contract  to  investigate 
the  transition  of  new  software  technology,  analyze  software  development  environments,  and 
provide  education  in  software  and  systems  engineering  processes.  As  part  of  the  contract,  the 
Software  Engineering  Institute  developed  the  software  capability  evaluation  to  assess  a 
contractor’s  strengths  and  weaknesses  in  software  development. 
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Laboratory  should  be  treated  as  a  Government  field  office  not  subject  to 
providing  software  metrics.  However,  the  Laboratory  is  not  a  Government 
entity  and  is  not  exempt  from  providing  software  metrics. 

Cost/Schedule  Control  System.  The  Navy  did  not  require  the  Laboratory  to 
provide  documentation  to  support  the  CEC  Program  cost,  schedule,  and 
performance  objectives.  The  DoD  Instruction  5000.2  requires  cost/schedule 
control  systems  criteria  on  research,  development,  test,  and  evaluation  contracts 
and  subcontracts  of  more  than  $60  million  in  FY  1990  dollars  for  all  acquisition 
programs,  including  highly  sensitive  classified  programs.  The  criteria  provide 
the  contractor  and  the  Government  program  managers  with  accurate  data  to 
monitor  execution  of  their  programs  and  an  adequate  basis  for  the  contractor 
and  the  Government  to  make  responsible  decisions. 

Officials  in  the  CEC  Program  Office  stated  that  they  would  have  liked  to  have 
received  cost/schedule  control  system  data  on  the  CEC  Program,  but  the  current 
SPAWAR  omnibus  contract  prevented  them  from  requesting  the  information 
from  the  Laboratory.  In  our  opinion,  the  CEC  Program  Office  must  obtain  this 
data  as  part  of  a  separate  engineering  and  manufacturing  development  contract 
with  the  Laboratory. 


Contract  Structure  and  Oversight  Responsibilities 


CEC  Program  Office  controls  to  manage  and  develop  the  CEC  Program 
effectively  were  inadequate  because  the  SPAWAR  contract  used  for  the 
development  of  the  CEC  Program  was  not  structured  for  the  development  and 
management  of  a  major  acquisition  category  IC  program. 

Contract  Structure.  The  PCO  and  SPAWAR  counsel  stated  that  they  were  not 
aware  that  the  omnibus  contract  included  taskings  for  an  ACAT  IC  program. 
They  believed  that  all  taskings  on  the  omnibus  contract  were  for  research  and 
development  efforts  and,  therefore,  did  not  structure  the  contract  to  obtain 
documentation  needed  to  manage  an  ACAT  IC  program.  After  they  became 
aware  that  the  CEC  Program  was  an  ACAT  IC  program,  they  agreed  that  the 
CEC  Program  Office  should  award  its  own  separate  contract  with  the 
Laboratory  for  the  CEC  Program. 


Effect  on  Contract  Management 


Without  controls  to  manage  the  CEC  Program  effectively,  the  CEC  Program 
Office  could  not  ensure  that  the  Laboratory  provided  fair  and  reasonable 
contract  prices  and  could  not  observe  and  measure  the  Laboratory's 
performance  against  contract  cost,  schedule,  and  performance  requirements. 
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Fair  and  Reasonable  Prices.  Since  the  Laboratory  did  not  provide  cost  and 
pricing  data  for  each  task  or  significant  subcontract,  the  PCO  could  not 
determine  whether  the  contract  tasking  prices  were  fair  and  reasonable. 

Performance  Measurement.  Because  the  omnibus  contact  was  not  structured 
to  manage  an  AC  AT  IC  program,  the  Program  Office  was  deprived  of  the 
necessary  base  for  program  and  technical  planning,  cost  estimating,  resource 
allocations,  performance  measurement,  and  status  reporting.  Further,  the  CEC 
Program  Office  could  not  verify  that  the  Laboratory  could  produce  a  system  that 
could  meet  users'  needs.  Without  receiving  cost/schedule  control  data,  a  work 
breakdown  structure,  a  clearly  defined  statement  of  work,  a  software 
development  plan,  and  software  metrics,  the  CEC  Program  Office  could  not 
validate  that  the  Laboratory's  work  progress  properly  related  cost,  schedule,  and 
technical  accomplishment.  Without  the  Laboratory  preparing  and  following  an 
approved  systems  engineering  management  plan,  the  CEC  Program  Office  could 
not  verify  that  development  decisions  were  based  on  operational  requirements. 
Without  a  quality  control  program,  the  CEC  Program  Office  did  not  have 
sufficient  information  to  determine  whether  the  Laboratory's  quality  control 
procedures  were  adequate,  reliable,  and  integrated  throughout  the  CEC 
Program,  including  the  systems  engineering  effort. 


Conclusion 


Since  the  CEC  Program  is  an  ACAT  IC  program,  the  Naval  Sea  Systems 
Command  should  issue  a  separate  contract  for  the  CEC  processor  and  be 
obtaining  information  necessary  to  manage  and  develop  the  CEC  Program 
effectively.  Documentation  requirements  in  the  CEC  processor  contract  should 
include  a  work  breakdown  structure,  a  clearly  defined  statement  of  work,  an 
approved  systems  engineering  management  plan,  an  adequate  quality  control 
program,  software  metrics,  and  cost/schedule  control  system  criteria.  Because 
SPAWAR's  omnibus  contract  was  not  intended  to  develop  an  ACATIC 
program,  the  Naval  Sea  Systems  Command  needs  to  award  a  separate  contract 
for  the  CEC  Program  during  the  engineering  and  manufacturing  development 
phase  of  the  acquisition  process  to  comply  with  system  acquisition  requirements 
in  DoD  Instruction  5000.2.  The  award  of  a  separate  engineering  and 
manufacturing  development  contract  would  enable  the  CEC  Program  Office  to 
manage  the  CEC  Program  properly  and  incorporate  cost  and  pricing  data 
requirements. 

After  we  completed  our  audit  fieldwork  and  staffed  our  working  draft  report, 
the  CEC  Program  Office  provided: 

o  a  draft  systems  engineering  management  plan,  dated  November  1994; 

o  summaries  of  software  trouble  reports;  and 

0  a  software  development  plan  for  the  CEC  processor,  dated  October 

1994. 


13 


Finding  A.  Program  Management  and  Development 


If  the  CEC  Program  Office  awards  the  CEC  processor  contract  to  a  contractor 
other  than  the  Laboratory,  the  new  contractor  will  have  to  prepare  a  software 
development  plan. 


Recommendations,  Management  Comments,  and  Audit 
Response 


We  recommend  that  the  Commander,  Naval  Sea  Systems  Command; 

1.  Award  a  separate  contract  for  the  Cooperative  Engagement 
Capability  processor  for  the  engineering  and  manufacturing  development 
phase  of  the  acquisition  process.  The  contract  should  include  requirements 
for  a  work  breakdown  structure,  a  clearly  defined  statement  of  work,  a 
systems  engineering  management  plan,  an  adequate  quality  control 
program,  a  software  development  plan,  software  metrics,  and  cost/schedule 
control  system  criteria  as  required  by  DoD  Instruction  5000.2,  "Defense 
Acquisition  Policies  and  Procedures,"  February  23, 1991. 

2.  Require  that  the  procuring  contracting  officer  assigned  to  the 
Cooperative  Engagement  Capability  processor  engineering  and 
manufacturing  development  contract  incorporate  cost  and  pricing  data 
requirements  into  the  contract  in  accordance  with  Federal  Acquisition 
Regulation,  subpart  15.8,  "Price  Negotiation." 

Management  Comments.  As  of  the  date  of  this  final  report,  we  had  not 
received  comments  from  the  Program  Manager,  CEC  Program,  to  a  draft  of  this 
report  issued  December  8,  1994.  We  requested  the  comments  by  February  6, 
1995. 

Audit  Response.  The  DoD  Directive  7650.3  requires  that  all  audit 
recoi^endations  be  resolved  promptly.  Therefore,  we  request  that  the  Navy 
provide  comments  on  this  final  report. 
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The  CEC  Program  Office  was  planning  to  declare  initial  operational 
capability  (IOC)  for  the  CEC  Program  in  FY  1996  based  on  an 
operational  assessment  of  prototype  CEC  equipment,  rather  than  on  an 
operational  test  of  production  CEC  equipment.  This  condition  occurred 
because  Congress  and  the  Navy  wanted  to  field  the  CEC  Program  as 
soon  as  possible  to  provide  Navy  ships  with  an  improved  anti-air  warfare 
capability.  In  addition,  the  condition  occurred  because  the  CEC 
Program  Office  did  not: 

0  include  and  fund  functional  and  structural  software  testing  and 
include  the  performance  of  a  dedicated  operational  test  and  evaluation  of 
a  production  CEC  configuration  in  the  draft  Test  and  Evaluation  Master 
Plan  (TEMP)  before  planning  to  declare  the  attainment  of  IOC; 

o  submit  the  revised  draft  TEMP  to  the  Director,  Test,  Systems 
Engineering  and  Evaluation  (DTSE&E),^  for  review  and  approval;  and 

0  work  effectively  with  the  DTSE&E  to  ensure  that  adequate 
CEC  developmental  test  results  would  be  available  before  IOC. 

As  a  result,  the  CEC  Program  Office  will  not  be  able  to  provide 
decisionmakers  with  adequate  developmental  and  operational  testing  data 
to  demonstrate  that  the  CEC  Program  is  operationally  effective  and 
suitable  and  satisfies  user  requirements  at  the  planned  IOC  date. 


Background 


DoD  Policy.  DoD  Instruction  5000.2  establishes  DoD  policy  for  conducting 
developmental  and  operational  testing  to  determine  whether  a  system  meets 
contract  technical  requirements  and  is  operationally  effective  and  suitable  before 
a  system  is  fielded.  In  this  regard,  the  Instruction  requires  that  TEMP  for  all 
acquisition  category  I  programs  be  approved  by  the  DTSE&E  and  the  Director, 
Operational  Test  and  Evaluation;  and  program  offices  grant  the  DTSE&E  and 
the  Director,  Operational  Test  and  Evaluation,  full  and  timely  access  to  all 
available  developmental  and  operational  test  information.  Additional  guidance 
concerning  developmental  and  operational  testing  is  in  Appendix  D. 

Initial  Operational  Capability.  DoD  Instruction  5000.2  defines  IOC  as  the 
first  attainment  of  the  capability  to  employ  effectively  a  system  of  approved 
characteristics  operated  by  a  trained,  equipped,  and  supported  military  unit  or 
force. 


^Before  November  1,  1994,  name  was  Director,  Test  and  Evaluation. 
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Status  of  Program  Requirements  and  Testing  Documentation.  As  of 
November  1994,  the  CEC  Pro^am  Office  did  not  have  an  approved  operational 
requirements  document  detailing  the  minimum  acceptable  CEC  operational 
requirements  or  a  TEMP  detailing  the  measures  of  effectiveness  with 
appropriate  quantitative  criteria  to  provide  substantive  evidence  for  analysis  of 
hardware,  software,  and  system  maturity  at  the  conclusion  of  testing.  The  CEC 
Program  Office  plans  to  have  the  two  documents  approved  before  the 
Milestone  II,  Development  Approval,  decision  scheduled  for  March  1995. 

Congressional  Direction  for  FY  1995.  The  CEC  Program  Office  interpreted 
the  Report  of  the  Committee  on  Appropriations  to  accompany  House  Report 
No.  4650,  "Department  of  Defense  Appropriations  Bill  1995"  (House  Report 
No.  103-562),  June  27,  1994,  to  require  the  Navy  to  field  the  CEC  by 
FY  1996.  Specifically,  the  Committee  Report  stated  that 

The  Committee  has  been  very  concerned  about  programs  to  protect 
Navy  ships  from  sea-skimming,  low-observable,  anti-ship  cruise 
missile  attack  since  the  time  when  37  sailors  died  in  the  attack  on  the 
U.S.S.  Stark.  The  Committee  commends  the  Assistant  Secretary  of 
the  Navy  for  Research,  Development,  and  Acquisition  for  her 
attention  to  these  programs  and  for  her  responsiveness  to  the 
Committee.  She  is  the  first  and  only  Defense  Department  witness  to 
testify  before  the  Committee  to  state  that  Congressional  direction  to 
field  certain  classified  capabilities  by  1996  will  be  accomplished.  In 
addition,  the  Navy  has  embarked  on  an  aggressive  program  to  field 
cooperative  engagement  capability  in  the  E-2  aircraft  much  earlier 
than  any  previous  plan. 


Adequacy  of  Operational  Assessment  Supporting  the 
Program's  Initial  Operational  Capability 


The  CEC  Program  Office  was  planning  to  use  an  operational  assessment  of  the 
CEC  prototype  to  support  and  declare  the  attainment  of  IOC  in  FY  1996  instead 
of  waiting  for  the  test  results  from  the  dedicated  operational  test  of  a  production 
CEC  test  article  to  be  completed  in  FY  1999.  The  CEC  Program  Office  was 
planning  to  declare  prematurely  the  attainment  of  IOC  for  the  CEC  in  FY  1996 
in  reaction  to  congressional  direction  that  required  the  Navy  to  field  the  CEC  by 
FY  1996.  The  IOC  declaration  will  be  premature  because; 

o  the  CEC  Program  Office  did  not  require  the  Laboratory  or  E-Systems 
to  do  CEC  prototype  fonctional  and  structural  software  testing  before  the  June 
1994  developmental  test  of  the  CEC  supporting  the  operational  assessment  and 

o  the  Commander,  Operational  Test  and  Evaluation  Force,  the  Navy's 
independent  operational  tester,  did  not  prepare  the  operational  assessment  to 
support  the  IOC  date  based  on  an  operational  test  of  a  representative  CEC 
system  configuration  tested  under  realistic  combat  conditions. 
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Further,  DoD  Instruction  5000.2  specifically  states  that  operational  assessments 
may  be  made  at  any  time  using  prototypes  but  will  not  substitute  for  the 
independent  operational  test  and  evaluation  necessary  to  support  decisions  to 
field  production  units  (emphasis  added). 

Functional  and  Structural  Software  Testing.  The  Defense  Systems 
Management  College  states  in  its  "Mission  Critical  Computer  Resources 
Management  Guide"  (undated)  that  functional  and  structural  software  testing  is 
done  to  determine  the  integrity  of  the  software.  The  Guide  states  that 
developmental  testers  should  verify  that  the  software  works  before  it  is 
integrated  into  the  hardware  because  integrating  untested  software  with  untested 
hardware  makes  it  difficult  to  determine  whether  a  program  malfunction  is 
hardware  or  software  related.  Functional  testing  entails  testing  a  software  unit 
with  valid  and  invalid  data  to  judge  the  quality  of  the  output  and  stressing  the 
software  to  failure  to  determine  the  ability  of  the  software  to  recover  from 
abnormal  events  or  to  fail  in  a  controlled  manner.  Structural  testing  entails 
testing  the  internal  logic  of  the  software  unit.  Specifically,  the  testing 
minimizes  errors  in  the  delivered  code  and  provides  reasonable  assurance  that 
the  software  performs  to  specifications  before  it  is  integrated  into  the  hardware. 

Although  the  Naval  Surface  Warfare  Center,  Dahlgren  Division,  submitted 
proposals  to  the  CEC  Program  Office  to  accomplish  this  essential  software 
testing,  the  CEC  Program  Office  had  not  required  the  Laboratory  or  E-Systems 
to  do  functional  or  structural  software  testing  of  the  CEC  software  before  the 
June  1994  developmental  test  of  the  CEC. 

Operational  Assessment.  The  June  1994  developmental  test  and  operational 
assessment  of  the  CEC  prototype  hardware  and  software  was  not  based  on  a 
representative  CEC  system  configuration  tested  under  realistic  combat 
conditions. 

Incomplete  System  Configuration.  A  CEC-configured  E-2C 
"Hawkeye"  aircraft,  a  critical  component  of  the  CEC  Program,  was  not 
available  to  enable  the  CEC  to  coordinate  all  force  radars,  including  radars  of 
ships  over-the-horizon,  into  a  single  real-time,  fire  control,  radar  picmre.  A 
production  CEC-configured  E-2C  "Hawkeye"  aircraft  will  not  be  available  for 
testing  until  FY  1999.  In  addition,  the  Navy  was  not  able  to  test  the  composite 
identification  software  adequately,  another  critical  component  of  the  CEC 
Program.  Composite  identification  software  allows  ships  with  CEC  to  use 
identification  friend-or-foe  sensor  readings.  The  Laboratory  stated  that  the 
composite  identification  software  could  not  be  adequately  tested  because  it  was 
not  interoperable  with  Link-11,  an  existing  Navy  command  and  control  system, 
through  which  the  Navy  processes  identification  friend-or-foe  data.  The 
existing  Link- 11  processor  cannot  handle  the  amoimt  of  data  that  the  CEC 
Program  provides.  Accordingly,  the  CEC  Program  Office  will  have  to  resolve 
the  data  processing  limitations  of  the  Link-11  processor  before  the  CEC- 
dedicated  operational  test  and  CEC  Program  fielding. 

Developmental  Test  Environment.  The  developmental  test  involved 
the  use  of  five  ships  of  the  Eisenhower  Battlegroup  equipped  with  upgraded 
CEC  prototype  equipment  and  a  P-3  Orion  aircraft  configured  with  upgraded 
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CEC  prototype  equipment  to  emulate  an  E-2C  "Hawkeye"  aircraft.  The  Navy 
intends  to  use  this  equipment  in  the  field  until  production  Common  Equipment 
Set  components  become  available.  When  compared  to  the  prototype  and  to 
meet  user  requirements,  the  Common  Equipment  Set  components  will  have  to 
be  substantially  smaller  with  increased  processing  power.  The  Common 
Equipment  Set  will  weigh  about  1,500  pounds  and  500  pounds  for  shipboard 
and  airborne  units,  respectively.  The  current  shipboard  and  airborne  CEC 
prototype  units  weigh  about  8,000  pounds  and  4,800  pounds,  respectively. 

Although  the  CEC  Program  Office  contended  that  the  conditions  of  the 
developmental  test  were  similar  to  the  requirements  of  an  operational  test, 
personnel  from  the  Office  of  the  Director,  Operational  Test  and  Evaluation, 
who  witnessed  the  test,  stated  that  they  did  not  regard  the  test  as  an  operational 
test.  Specifically,  they  stated  that  the  CEC  Program  Office  plaimed  Ae  test  so 
that: 


o  the  likelihood  of  problems  was  remote; 

o  the  contractors,  who  developed  the  CEC  Program,  participated 
extensively  in  the  testing;  and 

o  the  weather  at  the  test  site  was  conducive  to  the  assessment. 

Also,  Laboratory  and  Chief  of  Naval  Operations  personnel  stated  that  the  ship 
alignment  was  not  representative  of  a  typical  operational  scenario.  Further, 
personnel  from  the  Office  of  the  Director,  Operational  Test  and  Evaluation, 
concluded  that  the  assessment  was  useful  for  CEC  development,  but  it  was  not 
adequate  to  certify  that  the  CEC  Program  was  ready  for  operational  fielding. 


Test  and  Evaluation  Oversight 


Classified  Program  and  Milestone  Review.  The  DTSE&E  had  limited 
oversight  of  the  CEC  Program.  This  condition  occurred  because  the  CEC 
Program  was  a  classified  program  until  1993  and  had  not  previously  had  a 
formal  milestone  review  by  the  Navy  Acquisition  Executive.  By  not  having  a 
formal  milestone  review,  the  CEC  Program  did  not  have  an  approved  program 
baseline  and  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
had  not  required  the  CEC  Program  to  submit  quarterly  Defense  Acquisition 
Executive  Summary  reports  concerning  the  CEC  Program's  performance  against 
program  cost,  schedule,  and  performance  requirements. 

Test  and  Evaluation  Master  Plan  Approval.  Through  November  1994,  the 
Office  of  the  Director,  Operational  Test  and  Evaluation,  has  reviewed  the 
operational  requirements  document  and  the  draft  TEMP  for  the  CEC  Program. 
However,  the  Office  has  withheld  TEMP  approval  until  the  Navy  included 
adequate  measures  of  effectiveness  to  measure  the  success  of  planned 
operational  tests  in  FY  1999.  With  respect  to  CEC  tests,  the  CEC  Program 
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Office  stated  that  the  Office  of  Commander,  Operational  Test  and  Evaluation 
Force,  will  not  be  formally  involved  with  CEC  testing  until  the  FY  1999 
dedicated  operational  tests. 

Sufficient  Developmental  Testing.  The  Office  of  the  DTSE&E  did  not  learn 
of  the  existence  of  the  CEC  Program  until  1993.  Until  December  1993,  the 
Program  was  a  classified  program.  Later,  the  CEC  Program  Office  did  not 
allow  the  Office  of  Ae  DTSE&E  to  review  the  draft  TEMP  for  the  CEC 
Program  before  the  Milestone  II  decision  scheduled  for  March  1995  as  required 
by  DoD  Instruction  5000.2.  As  a  result,  personnel  in  the  Office  of  the 
DTSE&E  stated  that  they  did  not  know  enough  about  the  CEC  Program  to 
determine  whether  the  CEC  Program  Office  planned  to  accomplish  sufficient 
developmental  testing  to  justify  that  a  dedicated  phase  of  operational  testing  was 
warranted  in  FY  1999.  The  Office  of  the  DTSE&E  also  agreed  that  the  Navy 
should  conduct  formal  operational  tests  before  declaring  that  the  CEC  had 
attained  IOC. 


Effect  of  Developmental  and  Operational  Testing  on  the 
Planned  Initial  Operational  Capability  Date 


Since  the  CEC  Program  Office  plans  to  declare  the  attainment  of  IOC  for  the 
CEC  Program  based  on  an  operational  assessment  of  prototype  CEC  equipment, 
rather  than  on  an  operational  test  of  CEC  production  equipment,  Navy 
decisionmakers  will  not  have  adequate  developmental  and  operational  testing 
data  to  be  confident  that  the  CEC  will  meet  user  requirements  and  is 
operatiomlly  effective  and  suitable.  The  level  of  developmental  testing 
accomplished  will  preclude  Navy  decisionmakers  from  knowing  whether  the 
CEC  Program: 

o  has  achieved  contract  technical  performance  requirements  and 

o  is  ready  for  operational  test  and  evaluation  to  demonstrate  that  the 
CEC  Program  is  ready  for  fielding. 

The  absence  of  dedicated  operational  tests  will  preclude  Navy  decisionmakers 
from  knowing  whether  the  CEC  Program: 

o  is  operationally  effective  and  suitable, 

o  meets  the  overall  degree  of  mission  accomplishment  when  operated  by 
representative  personnel  in  the  environment  planned  or  expected  for  operational 
use,  and 

0  can  be  placed  satisfactorily  in  the  field. 
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Recommendations,  Management  Comments,  and  Audit 
Response 

1.  We  recommend  that  the  Program  Manager,  Cooperative  Engagement 
Capability  Program, 

a.  Revise  the  draft  Test  and  Evaluation  Master  Plan  to: 

(1)  Include  and  fund  functional  and  structural  software 
testing  before  the  dedicated  operational  test  and  evaluation  planned  in 
FY  1999. 


(2)  Include  the  performance  of  a  dedicated  operational  test 
and  evaluation  of  a  production  conflguration  of  the  Cooperative 
Engagement  Capability  Program  before  declaring  the  attainment  of  initial 
operational  capability. 

b.  Submit  the  revised  draft  Test  and  Evaluation  Master  Plan  to  the 
Director,  Test,  Systems  Engineering  and  Evaluation,  for  review  and 
approval  before  the  Milestone  n.  Development  Approval,  decision 
scheduled  for  March  1995  as  required  by  DoD  Instruction  5000.2,  "Defense 
Acquisition  Management  Policies  and  Procedures." 

Management  Comments.  As  of  the  date  of  this  final  report,  we  had  not 
received  comments  from  the  Program  Manager,  CEC  Program,  to  a  draft  of  this 
report  issued  December  8,  1994.  We  requested  the  comments  by  February  6, 
1995. 

Audit  Response.  The  DoD  Directive  7650.3  requires  that  all  audit 
recommendations  be  resolved  promptly.  Therefore,  we  request  that  the 
Program  Manager,  CEC  Program,  provide  comments  on  this  final  report. 

2.  We  reconunend  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition)  request  Congress  to  extend  the  date  for 
attainment  of  initial  operation^  capability  for  the  Cooperative  Engagement 
Capability  Program  to  a  date  where  the  attainment  can  be  based  on  an 
operational  test  of  production  representative  equipment  rather  than  an 
operational  assessment  of  incomplete  prototype  equipment. 

Management  Comments.  The  Director,  Operational  Test  and  Evaluation  (the 
Director),  stated  that  he  did  not  have  the  authority  to  implement  the  draft  report 
recommendation.  In  the  draft  report,  we  recommended  that  the  Director  not 
approve  the  TEMP  until  it  includes  the  performance  of  a  dedicated  operational 
test  and  evaluation  of  a  production  configuration  of  the  CEC  Program  before 
declaring  the  attainment  of  IOC.  Procedurally,  the  Director  stated  that  his 
approval  of  the  TEMP  was  based  on  provisions  in  the  TEMP  for  a  dedicated 
operational  test  and  evaluation  to  support  a  beyond  low-rate  initial  production 
and  not  a  declaration  of  the  attainment  of  IOC.  The  Director  further  stated  that. 
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procedurally,  the  Navy  decides  when  to  field  the  CEC  system  with  interim  or 
full  capability  and  declares  IOC.  The  complete  text  of  the  Director's  comments 
is  in  Part  IV. 

Audit  Response.  Based  on  the  Directors'  comments,  we  revised  and  redirected 
Recommendation  B. 2.  to  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition).  Since  the  Navy's  decision  to  declare  the 
attainment  of  IOC  on  an  operational  assessment  of  incomplete  prototype  CEC 
equipment  was  based  on  the  requirement  to  meet  a  congressionally  mandated 
FY  1996  IOC,  we  recommend  tiiat  the  Assistant  Secretary  request  relief  from 
Congress  so  that  the  IOC  attainment  date  can  be  based  on  an  operational  test  of 
production  CEC  equipment  rather  than  an  operational  assessment  of  incomplete 
prototype  CEC  equipment.  We  request  that  the  Navy  provide  comments  on  this 
final  report. 

3.  We  recommend  that  the  Director,  Test,  Systems  Engineering  and 
Evaluation,  evaluate  the  Cooperative  Engagement  Capability  Program's 
developmental  testing  results  to  determine  whether  the  Program  is  ready 
for  operational  testing. 

Management  Comments.  As  of  the  date  of  this  final  report,  we  had  not 
received  comments  from  the  Director,  Test,  Systems  Engineering  and 
Evaluation,  to  a  draft  of  this  report  issued  December  8,  1994.  We  requested  the 
comments  Ijy  February  6,  1995. 

Audit  Response.  The  DoD  Directive  7650.3  requires  that  all  audit 
recommendations  be  resolved  promptly.  Therefore,  we  request  that  the 
Director,  Test,  Systems  Engineering  and  Evaluation,  provide  comments  on  this 
final  report. 


This  page  was  left  out  of  orignial  document 
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Appendix  A.  Results  of  Hotline  Allegations 

Review 


We  substantiated  11  of  13  Hotline  assertions  and  allegations  concerning  the 
CEC  Program.  The  results  of  our  review  of  each  item  follow. 

Assertion  1.  The  Johns  Hopkins  University/Applied  Physics  Laboratory 
(the  Laboratory)  was  the  prime  contractor  for  the  CEC  ITogram  and  had 
the  authority  to  make  technical  program  decisions  concerning  the  Program, 
an  acquisition  category  (ACAT)  IC  program. 

Partially  Substantiated.  While  technically  not  the  prime  contractor  for  the 
CEC  Program,  the  Laboratory  oversees  engineering  and  development  for  the 
CEC  Program,  an  ACAT  IC  program.  The  DoD  Instruction  5000.2,  "Defense 
Acquisition  Policies  and  Procedures,"  February  23,  1991,  defines  a  prime 
contractor  as  a  contractor  having  responsibility  for  design  control  and  delivery 
of  a  system.  The  Laboratory  was  the  Cooperative  Engagement  Processor  (the 
CEC  processor)  design  agent  and  the  CEC  Program's  tech^cal  direction  agent. 

Design  Agent.  As  design  agent,  the  Laboratory  designed  and  developed 
the  CEC  processor  software,  the  most  critical  CEC  Program  objective.  As  the 
CEC  processor  developer,  the  Laboratory  has  approval  from  the  Office  of 
Management  and  Budget  to  act  as  a  commercial  contractor.  The  Laboratory's 
omnibus  contract  with  the  Navy  guarantees  profit  to  the  Laboratory  based  on  the 
amount  of  work  it  performs  for  Ihe  Government. 

Technical  Direction  Agent.  The  CEC  Program  Office  designated  the 
Laboratory  as  the  technical  direction  agent  to  make  technical  program  decisions 
concerning  the  CEC  Program.  This  designation  occurred  because  the  CEC 
Program  Office  did  not  have  the  expertise  to  make  technical  program  decisions. 
The  Naval  Sea  Systems  Command  Instruction  5400. 57A,  "Delegation  of 
Technical  Responsibility  and  Authority  to  Engineering  Agents,"  December  6, 
1985,  allows  a  program  office  to  designate  die  contractor  or  a  Government 
organization  as  the  technical  direction  agent.  In  our  discussions  with 
Laboratory  personnel,  they  stated  that  the  Laboratory  was  a  research  and 
development  facility  with  expertise  in  rapidly  producing  prototypes;  however, 
they  did  not  consider  themselves  suited  to  be  prime  contractors  for  ACAT  I 
weapon  systems. 

Allegation  2.  The  Laboratory  was  not  held  to  the  standards  required  of  a 
contractor  for  an  ACAT  IC  program. 

Substantiated.  The  CEC  Program  Office  made  the  Laboratory  the  CEC 
processor  software  contractor  without  requiring  it  to  adhere  to  DoD  acquisition 
regulations  for  an  ACAT  IC  program.  The  DoD  Instruction  5000.2  requires 
that  a  contractor  for  an  ACAT  IC  program: 

o  establish  a  work  breakdown  structure. 
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o  prepare  a  software  development  plan, 

0  prepare  cost  performance  measurement  reports, 

o  establish  software  metrics, 

0  adhere  to  a  clearly  defined  statement  of  work,  and 

o  have  a  documented  quality  control  program. 

The  Federal  Acquisition  Regulation  also  requires  contractors  to  provide  cost  and 
pricing  data  for  each  tasking  on  cost-type  contracts. 

The  CEC  Program  Office  did  not  require  the  Laboratory  to  comply  with  the 
above  requirements.  Conversely,  the  CEC  Program  Office  required  E-Systems, 
the  developer  of  the  CEC  Program's  Data  Distribution  System,  to  comply  with 
the  above  requirements. 

After  we  completed  our  fieldwork  and  staffed  our  working  draft  report,  the 
CEC  Program  Office  provided  a  software  development  plan  for  die  CEC 
processor  prepared  by  the  Laboratory.  The  plan  was  dated  October  1994. 

Allegation  3.  Laboratory  personnel  had  subcontracts  on  the  CEC 
Program. 

Unsubstantiated.  We  did  not  find  evidence  that  Laboratory  employees  held 
subcontracts  on  the  CEC  Program.  Further,  Laboratory  personnel  stated  that 
they  did  not  allow  this  practice,  which  is  against  their  code  of  ethics. 

Allegation  4.  The  Laboratory  was  not  providing  the  CEC  Program  Office 
with  adequate  documentation. 

Substantiated.  As  stated  in  Allegation  2,  the  Laboratory  was  not  providing  the 
CEC  Program  Office  with  adequate  documentation.  However,  the  CEC 
Program  Office's  administrative  letters  did  not  specify  a  requirement  for  the 
Laboratory  to  provide  the  documentation.  The  DoD  Instruction  5000.2  requires 
the  program  office  to  obtain  adequate  documentation  to  manage  a  major 
acquisition  program  effectively. 

Allegation  5.  The  Laboratory  did  not  prepare  a  CEC  processor  software 
development  plan. 

Substantiated.  The  Laboratory  did  not  have  a  formal  CEC  processor  software 
development  plan  until  October  1994.  The  software  development  plan  is  an 
agreement  between  a  program  office  and  the  contractor  stating  how  the 
contractor  will  develop  the  software.  The  software  development  plan  provides 
&e  program  office  with  a  baseline  to  measure  progress  or  deviations  that 
indicate  whether  the  contractor  is  experiencing  difficulty  in  developing  the 
software  or  may  be  developing  software  that  will  not  satisfy  user  requirements. 
Before  October  1994,  the  CEC  Program  Office  was  missing  a  key  management 
tool  to  evaluate  the  Laboratory's  perf^ormance  effectively. 
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Allegation  6.  The  CEC  Program  did  not  have  an  integrated  test  program 
schedule. 

Substantiated.  The  CEC  Program  Office  provided  insufficient  direction  to  the 
Laboratory  concerning  test  and  evaluation  procedures.  Provisions  in  the 
omnibus  contract  required  the  CEC  Program  Office  to  state  in  the  administrative 
letter  for  each  contract  tasking  any  testing  that  the  Laboratory  must  accomplish. 
Laboratory  personnel  stated  that  the  CEC  Program  Office  did  not  task  them  to 
establish  an  integrated  test  program  schedule  for  the  CEC  Program  because  the 
CEC  Program  Office  was  operating  under  a  schedule-driven  acquisition 
strategy.  Additionally,  the  Director,  Operational  Test  and  Evaluation,  and  the 
Director,  Test,  Systems  Engineering  and  Evaluation,  had  not  approved  the  CEC 
Program  Test  and  Evaluation  Master  Plan  (TEMP).  DoD  Instruction  5000.2 
states  that  a  program's  TEMP  generates  detailed  test  and  evaluation  plans  and 
ascertains  schedule  and  resource  implications  associated  with  the  test  and 
evaluation  program.  However,  as  of  the  Jime  1994  Operational  Assessment,  the 
CEC  Program  did  not  have  an  approved  TEMP.  The  Office  of  the  Director, 
Operational  Test  and  Evaluation,  stated  that  the  CEC  Program's  TEMP  lacked 
specific  measures  of  effectiveness  necessary  to  evaluate  the  CEC  Program's 
performance  when  tested.  The  Office  of  the  Director,  Test,  Systems 
Engineering  and  Evaluation,  stated  that  the  Navy  had  not  requested  it  review 
and  approve  the  draft  TEMP. 

Allegation  7.  The  CEC  Program  Office  had  not  reviewed  the  Laboratory's 
quality  control  procedures. 

Substantiated.  The  CEC  Program  Office  had  not  reviewed  the  Laboratory's 
quality  control  procedures  to  determine  whether  the  procedures  were  adequate. 
The  DoD  Instruction  5000.2  requires  the  program  office  to  emphasize  and 
integrate  quality  control  throughout  a  program  by  integrating  quality  control 
into  the  systems  engineering  effort.  As  a  result,  the  CEC  Program  Office  had 
not  assured  itself  that  the  Laboratory's  quality  control  procedures  were  adequate 
for  developing  the  CEC  processor  software. 

Allegation  8.  The  Laboratory  did  not  follow  DoD  acquisition  regulations. 

Substantiated.  See  discussion  for  Allegation  2.  Also,  Laboratory  personnel 
stated  that  they  did  not  envision  the  CEC  Program  becoming  an  acquisition 
program.  They  considered  the  Program  to  be  a  research  and  development 
effort.  As  a  result,  the  Laboratoiy  did  not  establish  procedures  to  adhere  to 
DoD  acquisition  regulations  pertaining  to  the  acquisition  of  weapon  systems. 
As  stated  in  Allegation  4,  the  CEC  Program  Office  was  responsible  for 
requiring  and  obtaining  acquisition  documentation  necessary  to  manage  the 
Program  effectively. 

Allegation  9.  The  Laboratory  did  not  allow  the  CEC  Program  Office  to 
observe  its  development  process. 
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Unsubstantiated.  The  CEC  Program  Office  observed  the  Laboratory's 
deployment  process;  however,  it  did  not  require  the  Laboratory  to  provide 
information  to  enable  the  CEC  Program  Office  to  make  fully  informed 
management  decisions. 

Allegation  10.  The  CEC  system  will  have  performance  shortfalls  when 
provided  to  the  Navy. 

Substantiated.  The  CEC  Program  Office  did  not  require  the  Laboratory  to 
conduct  any  Government  acceptance  testing  of  the  CEC  hardware  and  software 
and  stress  testing  of  the  CEC  software.  The  CEC  Program  Office  tasked  the 
Naval  Surface  Warfare  Center,  Dahlgren  Division  (die  Center),  the  CEC 
software  support  agent,  to  conduct  acceptance  and  stress  testing.  However,  the 
Center  did  not  conduct  the  tests  because  of  inadequate  funding  and  equipment. 
Further,  the  CEC  Program  Office  accomplished  operational  assessments  and 
other  developmental  tests  to  date  without  an  approved  TEMP.  As  a  result,  the 
CEC  Program  Office  cannot  be  certain  that  software  and  hardware  delivered  to 
the  Navy  will  be  free  from  defects  and  able  to  meet  user  requirements 
effectively  and  economically  when  the  Navy  plans  to  declare  the  attainment  of 
initial  operational  capability  for  the  CEC  Program  in  FY  1996. 

Allegation  11.  The  Laboratory  was  not  providing  the  CEC  Program  Office 
with  software  metrics. 

Substantiated.  The  CEC  Program  Office  has  not  required  that  the  Laboratory 
provide  software  metrics.  As  a  result,  the  Program  Office  does  not  have 
adequate  documentation  to  measure  the  Laboratory's  progress  in  developing  the 
CEC  processor  software. 

Allegation  12.  The  Laboratory  did  not  prepare  a  work  breakdown 
structure. 

Substantiated.  The  CEC  Program  Office  did  not  require  the  Laboratory  to 
develop  a  work  breakdown  structure  for  program  and  technical  planning,  cost 
estimating,  resource  allocations,  performance  measurement,  and  status 
reporting.  As  a  result,  the  Laboratory  did  not  prepare  a  work  breakdown 
structure. 

Allegation  13.  The  Defense  Contract  Audit  Agency  did  not  audit  the 
Laboratory  contract  for  the  CEC  Program. 

Partially  Substantiated.  The  Defense  Contract  Audit  Agency  was  on  site  at 
the  Laboratory;  however,  the  Defense  Contract  Audit  Agency  was  unable  to 
provide  sufficient  audit  coverage  because  of  the  lack  of  required  DoD 
acquisition  documentation,  such  as  contract  cost  and  pricing  data,  a  work 
breakdown  structure,  and  a  clearly  defined  statement  of  work. 
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Acquisition  Category  ID  and  IC.  These  acquisition  categories  (ACATs)  are 
assigned  to  major  defense  acquisition  programs  that  have  unique  statutorily 
imposed  acquisition  strategies,  execution,  and  reporting  requirements.  The 
programs  are  estimated  by  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  (the  Under  Secretary)  to  require  an  eventual  total  expenditure  for 
research,  development,  test,  and  evaluation  of  approximately  $300  million  in 
FY  1990  constant  dollars  and  procurement  of  approximately  $1.8  billion  in 
FY  1990  constant  dollars.  The  Under  Secretary  designates  the  programs  as 
either  AC  AT  ID,  making  himself  the  milestone  decision  authority,  or 
ACAT  IC,  making  the  cognizant  DoD  Component  Head  the  milestone  decision 
authority.  The  DoD  Component  Head  can  further  delegate  to  the  Component 
Acquisition  Executive  the  authority  as  the  milestone  decision  authority. 

Administrative  Letter.  Laboratory  and  program  office  modifications  to  the 
omnibus  contract  that  describe  the  work  and  the  cost  of  the  work. 

Approved  Program  Baseline.  An  approved  program  baseline  embodies  the 
cost,  schedule,  and  performance  objectives  for  the  program.  The  milestone 
decision  authority  approves  the  baseline  at  milestone  reviews.  Minimum 
acceptable  requirements  known  as  thresholds  accompany  the  objectives.  If 
program  results  do  not  meet  these  thresholds,  the  milestone  decision  authority 
requires  a  reevaluation  of  alternative  concepts  or  design  approaches.  At 
Milestone  II,  the  milestone  decision  authority  approves  the  development 
baseline  for  the  engineering  and  manufacturing  development  effort. 

Common  Equipment  Set.  The  Common  Equipment  Set  is  the  objective 
hardware  configuration  for  the  CEC  Program  consisting  of  the  CEC  antenna, 
CEC  processor,  and  the  Data  Distribution  System.  The  Common  Equipment 
Set  will  be  substantially  smaller  than  the  current  prototype  equipment  and  have 
more  processing  power.  The  Common  Equipment  Set  will  weigh  approximately 
1,500  pounds  and  500  pounds  for  shipboard  and  airborne  units,  respectively. 
The  current  shipboard  and  airborne  CEC  prototype  units  weigh  about 
8,000  pounds  and  4,800  pounds,  respectively. 

Cooperative  Engagement  Processor  and  Software.  The  Cooperative 
Engagement  Processor  and  software  allow  the  CEC  user  to  operationally 
execute  the  CEC  anti-air  warfare  capability. 

Cost/Schedule  Control  System  Criteria.  The  DoD  Instruction  5000.2  states 
that  the  purpose  of  cost,  schedule,  control  systems  criteria  is  to  provide  the 
contractor  and  Government  program  managers  with  accurate  data  to  monitor  the 
execution  of  their  programs  and  to  provide  an  adequate  basis  for  responsible 
decisionmaking  by  requiring  the  contractor's  internal  management  control 
systems  to  produce  data  that: 

o  indicate  work  progress; 

0  relate  cost,  schedule,  and  technical  accomplishment; 
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o  provide  valid,  timely,  and  auditable  information;  and 

o  provide  DoD  Component  managers  with  information  summarized  at  a 
practical  level. 

Further,  the  Instruction  requires  research  and  development  cost-type  contracts 
and  subcontracts,  totaling  $60  million  or  more,  to  have  cost/schedule  control 
systems  criteria.  The  Instruction  does  not  require  compliance  with 
cost/schedule  control  systems  criteria  on  firm  fixed-price  or  level-of-effort 
contracts;  however,  the  milestone  decision  authority  can  make  exceptions  to  the 
Instruction  when  deemed  necessary. 

Data  Distribution  System.  The  Data  Distribution  System  is  the 
communications  portion  of  the  CEC  Program.  It  is  comprised  of  hardware  and 
software  that  allow  CEC-equipped  units  to  share  the  information  processed  by 
the  CEC  processor. 

Defense  Acquisition.  The  DoD  Directive  5000.1  establishes  policies  for 
translating  operational  needs  into  stable,  affordable  programs;  acquiring  quality 
products;  and  organizing  for  efficiency  and  effectiveness.  The  Directive  states 
that  the  program  office  will  structure  the  acquisition  process  in  discrete  logical 
phases  separated  by  major  decision  points  called  milestones  and  acquisition 
strategies  are  to  be  event-driven  and  link  major  contractual  commitments  and 
milestones  decisions  to  demonstrated  accomplishments  in  development  and 
testing. 

Deploy /Deployment.  Deploy /Deployment  is  fielding  of  the  weapon  system  by 
placing  it  into  operational  use  with  trained  units  in  the  field  or  fleet. 

Design  Agent.  The  CEC  Program  Office  refers  to  its  contractors  (system 
developers)  as  design  agents. 

Integrated  Test  Plan.  An  integrated  test  plan  records  individual  test  plans  for 
the  subcontractor,  prime  contractor,  and  the  Government.  The  plan  includes  all 
developmental  tests  that  the  prime  contractor  and  the  subcontractor  perform  at 
the  system  and  subsystem  levels.  The  plan  is  a  detailed,  working-level 
document  that  assists  the  program  office  in  identifying  risk  as  well  as 
duplicative  or  missing  test  activities.  A  well-maintained  plan  facilitates  the  most 
efficient  use  of  test  resources. 

Integrated  Test  Program  Schedule.  An  integrated  test  program  schedule  is  a 
chart  in  a  program's  TEMP  that  displays  a  program's  time  sequencing  of  the 
critical  test  and  evaluation  phases,  related  activities,  and  planned  cumulative 
funding  expenditures  by  appropriation. 

Link-11.  Link-11  is  a  Navy  command  and  control  system  that  provides  for 
communication  between  ships. 
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Milestone  Decision  Authority.  At  each  decision  point,  the  milestone  decision 
authority  for  an  ACAT  ID  and  IC  pro^am  assesses  the  status  of  the  program 
relative  to  the  user's  needs,  the  established  program  baseline  and  acquisition 
strategy,  and  approved  financial  plans.  The  milestone  decision  authority  also: 

o  evaluates  the  updated  acquisition  strategy  and  the  plans  for  conducting 
the  next  phase  and  managing  risk; 

0  makes  cost-performance-schedule  trade-offs; 

o  assesses  the  affordability  of  what  is  being  proposed;  and 

0  determines  whether  the  program  should  be  terminated,  redirected,  or 
allowed  to  continue  into  the  next  phase. 

Milestone  Decision  Points.  At  milestone  decision  points,  the  program  office 
provides  the  milestone  decision  authority  information  concerning  the  readiness 
of  the  program  to  proceed  to  the  next  phase  of  the  acquisition  process.  At: 

o  Milestone  I,  Concept  Demonstration  and  Approval,  the  milestone 
decision  authority  determines  whether  a  program  office  should  start  a  new 
acquisition  program; 

o  Milestone  II,  Development  Approval,  the  milestone  decision  authority 
determines  whether  the  program  office  should  continue  a  demonstration  and 
validation  program; 

o  Milestone  III,  Production  Approval,  the  milestone  decision  authority 
determines  whether  the  program  office  should  build,  deploy,  and  support  a 
program;  and 

o  Milestone  IV,  Major  Modification  Approval,  the  milestone  decision 
authority  determines  whether  the  system  currently  in  production  warrants  major 
modifications. 

Omnibus  Contract.  This  legal  agreement  between  the  Space  and  Naval 
Warfare  Systems  Command  and  the  Laboratory  provides  for  the  Laboratory  to 
develop  286  items  concurrently  for  the  Government. 

Operational  Requirements  Document.  An  operational  requirements  document 
is  a  document  containing  operational  effectiveness  and  operational  suitability 
parameters  for  a  proposed  system  or  concept. 

Software  Development  Plan.  A  software  development  plan  defines  the 
software  processes,  including  corporate  policies,  practices,  and  standards  that 
the  contractor  must  follow  throughout  the  software  development  process.  The 
Govenunent  must  verify  that  the  developer  understands  the  scope  of  the 
software  development  effort  and  that  the  developer  can  meet  the  user’s  needs. 

Systems  Engineering.  Systems  engineering  establishes  the  basis  for  integrating 
the  technical  efforts  of  the  design  team  to  meet  program  cost,  schedule,  and 
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performance  objectives.  The  objectives  include  an  optimal  design  solution  that 
encompasses  the  system  and  its  associated  manufacturing,  test,  and  support 
processes. 

Systems  Integration.  Systems  integration  is  the  engineering  efforts  needed  so  a 
weapon  system  will  function  effectively  with  other  existing  weapon  systems. 

Technical  Direction  Agent.  The  technical  direction  agent  is  a  contractor  that 
assists  the  program  office  in: 

o  establishing  initial  program  concepts; 

o  performing  or  directing  research,  development,  tests,  and  simulations 
to  investigate  problems; 

0  probing  alternative  technical  approaches;  and 

o  evaluating  design  agent  achievements. 

Test  and  Evaluation  Master  Plan.  A  test  and  evaluation  master  plan 
documents  the  overall  structure  and  objectives  of  the  test  and  evaluation 
program. 
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The  Inspector  General,  DoD,  issued  a  report  concerning  the  Johns  Hopkins 
University/Applied  Physics  Laboratory  (the  Laboratory).  Report  No.  95-001, 
"Navy  Proposed  Follow-On  Research  and  Development  Contract  for  Johns 
Hopkins  University  Applied  Physics  Laboratory,"  October  3,  1994,  stated  that 
the  Navy  intended  to  award  a  $1.2  billion  contract  noncompetitively  to  the 
Laboratory  without  adequate  justification.  The  task  order  structure  of  the 
proposed  contract  did  not  require  task  sponsors  to  seek  competition  for  the 
individual  task  orders  issued  under  the  contract  and  caused  management  and 
control  problems  for  contracting  personnel  and  other  oversight  groups. 
Additionally,  the  Navy  had  not  evaluated  the  use  of  the  fee*  paid  to  the 
Laboratory  since  1962.  The  report  reconunended  that  the  Navy: 

0  clearly  define  the  essential  capabilities  that  the  Navy  wanted  to 
maintain  at  the  Laboratory, 

o  demonstrate  that  the  Laboratory  was  uniquely  qualified  to  provide 
those  capabilities, 

o  determine  whether  sources  other  than  the  Laboratory  could  provide 
the  services  being  procured  from  the  Laboratory, 

0  prepare  a  basic  ordering  agreement  to  replace  the  task  order  contract 
with  the  Laboratory,  and 

o  reassess  the  fee  arrangement  with  the  Laboratory. 

In  response  to  the  recommendations,  the  Navy  stated  that  it  intended  to: 

o  conduct  a  study  to  determine  whether  other  organizations  could 
provide  the  same  types  of  services  as  those  obtained  from  the  Laboratory, 

o  award  follow-on  contacts  in  the  basic  ordering  agreement  format  to 
other  smaller  university-affiliated  laboratories, 

0  transition  the  Laboratory's  contract  to  a  basic  ordering  agreement,  and 

o  assess  the  fee  paid  to  the  Laboratory. 

In  the  final  report,  the  Inspector  General,  DoD,  stated  that  the  Navy  comments 
were  responsive  to  the  recommendations  and  requested  that  the  Navy  provide 
the  completion  dates  for  planned  action  by  December  5,  1994. 


The  fee  establishes  a  stabilization  and  contingency  fund  to  provide  the  Laboratory  staff  with 
stable  funding,  to  ensure  reasonable  continuity  in  the  event  that  the  relationship  between  the 
Government  and  the  Laboratory  substantially  changes,  to  pay  management  costs  and  non¬ 
reimbursable  items,  to  protect  against  major  contract  disallowances,  and  to  reimburse  the  Johns 
Hopkins  University  for  costs  incidental  to  the  operation  of  the  Laboratory. 
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Developmental  Testing.  DoD  Instruction  5000.2,  "Defense  Acquisition 
Policies  and  Procedures,"  February  23,  1991,  requires  program  offices  to 
structure  developmental  test  and  evaluation  programs  in  the  TEMP  that  are 
designed  to  identify  potential  operational  and  technical  limitations  of  a  program 
and  to  substantiate  the  achievement  of  contract  technical  requirements. 

Operational  Testing.  DoD  Instruction  5000.2  requires  program  offices  to 
structure  operational  and  evaluation  programs  in  the  TEMP  that  are  designed  to 
determine  the  operational  effectiveness  and  suitability  of  a  weapon  system  under 
realistic  combat  conditions  and  to  determine  whether  the  weapon  system  will 
satisfy  the  minimum  acceptable  operational  requirements  as  specified  in  the 
operational  requirements  document. 

Operational  Assessments.  DoD  Instruction  5000.2  requires  the  Military 
Department's  independent  test  organization  to  conduct  an  early  operational 
assessment  in  support  of  the  Milestone  11 ,  Development  Approval,  decision. 
The  purpose  of  the  early  operational  assessment  is  to  note  significant  trends  in 
development  efforts,  areas  of  program  risk,  adequacy  of  requirements,  and  the 
ability  of  the  program  to  support  adequate  operational  testing.  The  Instruction 
specifically  states  that  operational  assessments  may  be  made  at  any  time  using 
prototypes  but  will  not  substitute  for  the  independent  operational  test  ar^ 
evaluation  necessary  to  support  full  production  decisions  (emphasis  added). 

Operational  Test  and  Evaluation.  Before  the  full  production  decision,  DoD 
Instruction  5000.2  requires  the  Military  Department's  independent  test 
organization  to  conduct  a  dedicated  phase  of  operational  test  and  evaluation 
(dedicated  operational  test)  using  production  test  articles.  The  use  of  system 
contractors  in  support  of  the  dedicated  operational  test  is  restricted  by  Title  10, 
United  States  Code,  Section  2399,  "Operational  Test  and  Evaluation  of  Defense 
Acquisition  Programs."  The  purpose  of  the  dedicated  operational  test  is  to 
determine  whether  the  weapon  system  is  operationally  effective  and  suitable 
under  realistic  combat  conditions. 
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Recommendation 

Reference 


A.l. 


A.2. 


B.l.a. 


B.l.b. 


B.2. 


B.3. 


Description  of  Beneflt 


Internal  Control  and  Program 
Results.  Will  provide  effective 
management  of  the  CEC  Program. 


Internal  Control.  Will  incorporate 
cost  and  pricing  data  into  the  CEC 
engineering  and  manufacturing 
development  contract. 

Internal  Control  and  Program 
Results.  Will  ensure  that  the  Test 
and  Evaluation  Master  Plan  for  the 
CEC  Program  includes  and  funds 
functional  and  structural  software 
testing  and  includes  the  performance 
of  a  dedicated  operational  test  and 
evaluation  before  the  initial 
operational  capability  date. 

Internal  Control  and  Program 
Results.  Will  ensure  that  the  Test 
and  Evaluation  Master  Plan  is 
submitted  for  review  and  approval 
before  the  Milestone  II, 
Development  Approval,  decision. 

Internal  Control  and  Program 
Results.  Will  ensure  that  the  CEC 
Program  is  operationally  effective 
and  suitable  before  the  Navy 
declares  the  CEC  Program  initial 
operational  capability  date. 

Internal  Control  and  Program 
Results.  Will  ensure  that 
development  test  results  support  that 
the  CEC  Program  is  ready  for 
operational  testing. 


Amount  and/or 
Type  of  BeneHt 


Nonquantifiable 
because  benefits 
depend  on  future 
Navy  actions. 

Nonquantifiable 
because  benefits 
depend  on  future 
Navy  actions. 

Nonquantifiable 
because  benefits 
depend  on  future  OSD 
and  Navy  actions. 


Nonquantifiable 
because  benefits 
depend  on  future  OSD 
and  Navy  actions. 


Nonquantifiable 
because  benefits 
depend  on  future 
Navy  actions. 


Nonquantifiable 
because  benefits 
depend  on  future  OSD 
and  Navy  actions. 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Washington,  DC 
Director,  Test,  Systems  Engineering  and  Evaluation,  Washington,  DC 
Under  Secretary  of  Defense  (Comptroller),  Washington,  DC 
Director,  Program  Analysis  and  Evaluation,  Washington,  DC 
Assistant  Secretary  of  Defense  (Command,  Control,  Communication  and  Evaluation), 
Washington,  DC 

Director,  Operational  Test  and  Evaluation,  Washington,  DC 


Department  of  the  Navy 

Chief  of  Naval  Operations,  Washington,  DC 

Cormnander  in  Chief,  U.  S.  Atlantic  Command,  Norfolk,  VA 

Assistant  Secretary  of  the  Navy  (Financial  Management),  Washington,  DC 

Naval  Sea  Systems  Command,  Arlington,  VA 

Cooperative  Engagement  Capability  Program  Office,  Arlington,  VA 
Naval  Surface  Warfare  Center,  D^gren  Division,  Dahlgren,  VA 
Space  and  Naval  Warfare  Systems  Command,  Arlington,  VA 

Naval  Technical  Representative  Office,  Applied  Physics  Laboratory,  Laurel,  MD 
Office  of  Naval  Research,  Arlington,  VA 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition),  Washington,  DC 


Other  Defense  Organizations 

Defense  Contract  Audit  Agency,  Alexandria,  VA 

Defense  Contract  Audit  Agency,  Applied  Physics  Laboratory,  Laurel,  MD 
Defense  Contract  Audit  Agency,  E-Systems,  ECI  Division,  Saint  Petersburg,  FL 
Defense  Contract  Audit  Agency,  Lockheed  Aeronautical  Systems  Company, 
Calabasas,  CA 

Defense  Contract  Audit  Agency,  Lockheed  Aeronautical  Systems  Company, 
Marietta,  GA 

Defense  Logistics  Agency,  Alexandria,  VA 

Defense  Contract  Management  Command,  Alexandria,  VA 
Defense  Contract  Management  Area  Operations,  Clearwater,  FL 
Director,  Joint  Staff,  Washington,  DC 
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Non-Defense  Federal  Organizations 

U.S.  Department  of  Transportation,  Washington,  DC 
U.S.  Department  of  Treasury,  Washington,  DC 

Contractors 

E-Systems,  ECI  Division,  Saint  Petersburg,  FL 

Johns  Hopkins  University /Applied  Physics  Laboratory,  Laurel,  MD 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 

Principal  Deputy  Under  Secretary  of  Defense  (Acquisition  and  Technology) 
Director,  Acquisition  Program  Integration 
Director,  Test,  Systems  Engineering  and  Evaluation 
Deputy  Under  Secretary  of  Defense  (Acquisition  Reform) 

Under  Secretary  of  Defense  (Comptroller) 

Director,  Program  Analysis  and  Evaluation 

Assistant  Secreta^  of  Defense  (Command,  Control,  Communications  and  Intelligence) 
Director,  Operational  Test  and  Evaluation 
Assistant  to  the  Secretary  of  Defense  (Public  Affairs) 

Director,  Administration  and  Management 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Secretary  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management) 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 
Comptroller  of  the  Navy 
Naval  Air  Systems  Command 
Naval  Sea  Systems  Command 

Cooperative  Engagement  Capability  Program  Office 
Naval  Surface  Warfare  Center,  Dahlgren  Division 
Space  and  Naval  Warfare  Systems  Command 

Naval  Technical  Representative  Office,  Applied  Physics  Laboratory 
Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 
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Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 

Defense  Contract  Audit  Agency,  Applied  Physics  Laboratory,  Laurel,  MD 
Director,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 

Commander,  Defense  Contract  Management  Command 

Defense  Contract  Management  Area  Operations,  Clearwater,  FL 
Director,  National  Security  Agency 
Director,  Joint  Staff 

Inspector  General,  Central  Imagery  Office 

Inspector  General,  National  Security  Agency 

Director,  Defense  Logistics  Studies  Information  Exchange 


Non-Defense  Federal  Organizations 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 

Chairman  and  Ranking  Minority  Member  of  Each  of  the  Following  Congressional 
Committees  and  Subcommittees; 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  National  Security 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
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Director,  Operational  Test  and  Evaluation, 
Comments 


OFFICE  OF  THE  SECRETARY  OF  DEFENSE 
WASHINGTON.  DC  2030 J-1  700 


OPCRATiONAU  rCS"*^  '  ^ 

AND  SVauUAT’ON 


MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  INSPECTOR  GENEEIAL 

SUBJECT:  Draft  of  Proposed  Audit  Report  on  the  Cooperative 
Engagement  Capability  (CEC)  Program 


Thank  you  for  the  opportunity  to  review  and  comment  on  the 
findings  and  reconraendations  in  your  proposed  report.  The  report 
findings  address  issues  in  two  areas:  (1}  Finding  A,  program 
management  and  development  and  (2)  Finding  B,  initial  operational 
capability  (IOC).  I  have  no  comment  regarding  Finding  A  and 
associated  recommendations . 

With  respect  to  Finding  B  and  associated  recommendations,  I 
have  provided  detailed  coiiments  below.  Please  note  ity  comment  on 
the  portion  of  the  recommendation  that  "the  Director.  Operational 
Test  and  Evaluation,  not  approve  the  Test  and  Evaluation  Master 
Plan  unless  it  includes  the  performance  of  a  ded.icated 
operational  test  and  evaluation  of  a  production-representative 
configuration  of  the  Cooperative  Engagement  Capability  Program 
before  declaring  the  attainment  of  (IOC)  .  •* 

There  are  several  criteria  that  must  be  satisfied  before  I 
will  approve  a  TEMP.  Among  these  is  adequate  provision  for 
dedicated  OT&E  prior  to  the  beyond  low- rate  initial  production 
[B-LRIP)  decision.  That  OT£E  serves  as  the  basis  for  my  B-LRIP 
Report  to  the  Congress,  which  must  precede  the  b-lrip  decision. 
There  is  no  TEMP  approval  criterion  keyed  to  declacafcion  of  IOC 
by  a  Service.  My  detailed  comments  are  attached. 


Philip  E.  Coyle 
Director 


Attachment 
As  stated 
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Director,  Operational  Test  and  Eyaluation,  Conunents 


COMMENTS  ON  DOD  16  WORKING  DRAFT 
OF  A  PROPOSED  AUDIT  REPORT  ON  THE 
COOPERATIVE  ENGAGEMENT  CAPABILITY  (CEO  PROGRAM 


The  subject  report  presents  two  findings  with  associated 
conclusions  and  recommendations.  The  report  findings  address 
issues  in  two  areas;  (1>  Finding  A,  program  management  and 
development  and  (2)  Finding  B,  initial  operational  capability 
(IOC).  I  have  no  comment  regarding  Finding  A  and  associated 
recommendations.  The  following  comments  are  with  regard  to  test 
and  evaluation  matters  associated  with  Finding  B  and  its 
associated  recornmendations . 

1,  Page  16,  non  policy.  It  is  stated  in  the  report  that  DoD 
Instruction  5000.2  established  DoD  policy  for  conducting 
developmental  and  operational  testing  to  determine  whether  a 
system  meets  contract  technical  requirements  and  is  operationally 
effective  and  suitable  before  a  system  is  fielded.  Coalmen t; 

DoDI  5000.2  establishes  defense  acquisition  management  policies 
and  procedures.  It  states  that  test  and  evaluation  programs 
shall  be  structured  to  ‘provide  essential  information  for 
assessment  of  acquisition  risk  and  for  decisionmaking*  and  to 
"verify  that  systems  are  operationally  effective  and  suitable  for 
intended  use."  Operational  test  and  evaluation  (OT&E)  is  a 
process  to  provide  acquisition  executives  an  independent 
assessment  of  operational  effectiveness  and  suitability  prior  to 
their  deciding  on  whether  or  not  to  commit  to  low-rate  initial 
production  or  full  rate  production.  Further,  I  report  to  the 
Congress,  based  on  the  results  of  dedicated  OT&E,  whether  the 
testing  was  adequate  and  whether  the  system  under  test  is 
operationally  effective  and  operationally  suitable,  prior  to  the 
decision  to  proceed  beyond  low- rate  initial  production.  The 
fielding  of  a  military  system  with  interim  or  full  capability  is 
a  Service  decision. 

2.  Page  18.  According  to  the  report,  DoDI  5000.2  states  that 
operational  assessments  may  be  made  at  any  time  using  prototypes 
but  will  not  substitute  for  the  independent  operational  test  and 
evalaution  necessary  to  support  decisions  to  field  production 
units,  comment ;  DoDI  5000.2,  Part  8  Test  and  Evaluation,  states 
(1)  that  OT<iE  does  not  include  an  operational  assessment  based 
exclusively  on  computer  modeling,  simulation  or  an  analysis  of 
certain  information  contained  in  program  documents  (Section  2,b); 
(2}  that  early  operational  assessments  shall  be  emphasized  to 
assist  in  identifying  risks  (Section  2.e);  and  (3)  that  a  final 
independent  phase  of  operational  test  and  evaluation  shall  be 
required  for  beyond  low-rate  production  decisions  (Section 

2. h. (2)). 

3,  Page  19.  According  to  the  report,  under  Developmental  Test 
Environment,  personnel  from  the  Office  of  Che  Director, 
Operational  Test  and  Evaluation,  stated  that,  the  CEC  Program 


Final  Report 
Reference 


15 


17 


17-18 
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Director,  Operational  Test  and  Evaluation,  Conunents 


Final  Report 
Reference 


18 


20-21 


Operational  Test  and  ETvaluation,  stated  that,  the  CEC  Program 
Office  planned  the  test  so  that:  (1)  the  weather  at  the  test 
site  was  conducive  to  the  assessment;  and  {2)  the  assessment  was 
not  adequate  to  certify  that  the  CEC  Program  was  ready  for 
operational  fielding.  Comment!  (1)  The  DOT&E  staff  assistant 
stated  that  tests  were  not  conducted  during  a  period  of  heavy 
rainfall  that  occurred  during  the  test  period  and  that  the 
Caribbean  Sea  environment  is  characterized  by  ducting,  which 
facilitates  data  link  corranunication  at  extended  distances.  {2} 
The  DOT&E  Staff  assistant  stated  that  the  assessment  was  not 
adequate  for  reaching  conclusions  on  operational  effectiveness 
and  operational  suitability  of  the  CEC  system. 

4.  Page  20.  According  to  the  report,  under  T«st  and  Evaluation. 
Master  Plan  Approval,  the  Office  of  Director,  Operational  Test 
and  Evaluation  has  reviewed  the  draft  TEMP  and  has  withheld  TEMP 
approval  until  the  Navy  included  adequate  measures  of 
effectiveness  to  measure  the  success  of  planned  operational  tests 
in  FY  1997,  Comment  r  Provision  of  adequate  measures  of 
effectiveness  is  one  of  several  criteria  considered  during  TEMP 
review.  A  Navy-approved  TEMP  for  CEC  has  not  been  submitted  for 
review  by  the  Office  of  the  Secretary  of  Defense. 

5.  Pages  21-22.  The  report  recommends  that  the  Director, 
Operational  Test  and  Evaluation,  not  approve  the  TEMP  unless  it 
includes  the  performance  of  a  dedicated  operational  test  and 
evaluation  of  a  product ion- representative  configuration  of  the 
CEC  Program  before  declaring  the  attainment  of  initial 
operational  capability,  Response ;  Adequate  provision  in  a  TEMP 
for  dedicated  OT&E  to  support  a  beyond  low-rate  initial 
production  (B-LRIP)  decision  is  among  the  criteria  that  must  be 
met  before  i  will  approve  a  TEMP.  There  is  no  temp  approval 
criterion  keyed  to  declaration  by  a  Service  of  partial  capability 
or  full  capability. 
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